DOCUMENT RESUME 



ED 448 964 



RC 022 724 



AUTHOR 

TITLE 

INSTITUTION 

PUB DATE 
NOTE 
PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Mearns , Curt 

Title IX Program Evaluation: Corresponding with FY 98 
Application for Grants. Albuquerque Public Schools, 1997-98. 
Albuquerque Public Schools, NM. Research, Development, and 
Accountability. 

1999-09-00 

12p . 

Numerical /Quantitative Data (110) -- Reports - Evaluative 

(142) 

MF01/PC01 Plus Postage. 

Academic Achievement; *American Indian Education; Cultural 
Awareness; Elementary Secondary Education; * Intervent ion; 
♦Parent Participation; *Professional Development; *Program 
Effectiveness; Program Evaluation; Service Learning; 
♦Tutoring 

♦Albuquerque Public Schools NM; American Indian Students; 
Johnson O Malley Act; Title IX Education Amendments 1972 



ABSTRACT 



Title IX and Johnson O’Malley services were provided to 
1,495 Native American students at 10 Albuquerque (New Mexico) elementary, 
middle, and high schools and the Homework Center during the 1997-98 school 
year. The five goals of the Title IX program were to provide: (1) 

intervention assistance for students, in which students, parents, and staff 
set goals in areas such as attendance and behavior problems; (2) tutoring 
assistance, which focused on language arts and math at the elementary level 
and added science and social studies at the secondary level; (3) student 
participation in cultural, social, or academic activities to promote good 
citizenship, community service, and personal responsibility; (4) parent 
participation in cultural, social, or academic activities, promoted through 
announcements in newsletters, letters sent home with students, and 
announcements at parent meetings; and (5) professional development in 
sensitive cultural issues that impact Native American student performance. In 
intervention and tutoring assistance, each school level surpassed its goal. 
The homework center was most effective, followed by assistance at the middle 
and elementary school levels. Students reported 99.5 percent of the time that 
the social, cultural, and academic activities were beneficial. Because of low 
survey response rates, parent participation was difficult to evaluate, but 88 
percent of the 18 respondents thought the activities were beneficial to 
students. Ninety-eight percent of respondents thought that the professional 
development was somewhat beneficial. (TD) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



Os 

00 



Q 

w 



Title IX Program Evaluation: 

Corresponding with FY 98 Application for Grants 

Albuquerque Public Schools 
1997-98 



r 



119 DEPARTMENT OF education 

Office^ of Ed ucational Research and Improvement 

ducational resources information 
7 CENTER (ERIC) 

'This document has been reproduced as 
received from the person or organization 
originating it. 

Minor changes have been made to 






FRMISSION to reproduce THIS 



_£2iurt— - 






• Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 




TO THE EDUCATIONAL RESOURCES \ 
,NF0RMATI0N CENTER (ERIC).” 



-st :• 

04 

Curt Mearns 
t '* September 1999 



BEST COPY AVAILABLE 




ALBUQUERQUE PUBLIC SCHOOLS 



BOARD OF EDUCATION 


RICHARD TOLEDO 


President 


AGATHA M. LOPEZ 


BILL C. ROTHANBARGAR 


Vice-President 


Secretary 


MATTHEW E. ARCHULETA. 


LEONARD J. DELAYO, JR. 


Member 


Member v 


DOLORES HERRERA 


MARY LEE MARTIN 


Member 


Member 



BRADFORD ALLISON, Ph.D. 
Superintendent 

JOSEPH M. VIGIL 
Associate Superintendent 



O 

ERIC 



RESEARCH, DEVELOPMENT AND ACCOUNTABILITY 

930-A Oak Street SE 
Albuquerque, New Mexico 87 106 
(505) 764-9710 

Rose- Ann McKernan . 

' “ ' ? ’ Directdf'^ ’ " ! - J • 



3 BEST COPY AVAILABLE: 



1997-1998 TITLE IX EVALUATION REPORT 



Albuquerque Public Schools Indian Education unit is responsible for 4,498 Native American 
students. The Title IX program operates under the supervision of the APS Indian Education unit. 
In the 1997-1998 school year, Title IX and Johnson O’Malley program services were provided to 
1,495 eligible students. Remaining Native American students were served through other means. 
Eleven staff members provided Title IX and Johnson O’Malley services to eligible students at 
the school sites or the Homework Center. Participating schools included: Whittier, La Mesa, 
Emerson, Hayes, Van Buren, Wilson, Rio Grande, West Mesa, Albuquerque, Highland. The 
Homework Center operated in the Montgomery Complex. 

METHODS 

Title IX staff members maintain a master list of all Native American students within the district. 
Additionally, a quarterly record is kept for all students who are provided service via intervention 
or tutoring goals. Within the quarterly record, the teacher enters student progress on the project- 
developed teacher rating scale. Finally, a yearly record is kept for assessing student progress via 
their grade point average. 



LIMITATIONS 

Title IX staff have meticulously maintained paper records and computer records. There are three 
sets of records; a master list, quarterly reports and a yearly report. Currently there is no way to 
reliably merge data from the three sets of records used to track student progress. Therefore, 
summaries necessarily reflect duplicate student records. 

GOALS 

There are five goals of the Title IX program as presented in the FY 98 grant application: 1) 
provide intervention assistance for students, 2) provide tutoring assistance, 3) provide 
opportunities for student participation in cultural and social activities, 4) provide opportunities 
for parents’ participation in cultural and social activities, 5) provide professional development 
opportunities for APS staff. 

I. “Provide intervention assistance for students” 

Intervention assistance (facilitation) involves students, parents and district personnel in 
collaboratively finding solutions to nonacademic, educationally related issues such as: medical 
issues, attendance problems, and behavior. Students, staff and parents agree upon goals that 
Title IX staff monitor on a quarterly basis using a teacher implemented rating scale as well as 
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quarterly grade point averages. Throughout the school year, Indian Education staff and 
classroom teachers engage students in activities designed to help the students achieve their goals. 

II. “Provide tutoring assistance as an intervention” 

Tutoring assistance is designed to improve the performance of Native American students at all 
levels of ability. Tutoring assistance generally follows district goals. At elementary schools, 
tutoring assistance focuses on language arts and math, while in secondary schools the focus 
broadens to include science and social studies. Students may receive instruction individually or 
in small groups within their classroom setting or in a separate setting (pullout style). Title IX 
staff members monitor tutoring goals using the quarterly teacher rating scale and quarterly grade 
point averages. Staff provided academic encouragement for finishing homework, clarification of 
classroom teacher’s instructions, or facilitation in procuring materials. An additional mechanism 
of providing academic assistance is the Homework Center, which is available to any eligible 
student from pre-kindergarten through 12 th grade. The Homework Center was available 
Tuesdays and Thursdays after school until 8pm and was staffed by a certified teacher and at 
times parent volunteers. Computers with educational enrichment software, internet access, 
multimedia applications and standard office software are available for student use. 

Title IX staff members used the same method of tracking the intervention and tutoring goals, so 
the results of both goals will be presented in this section. First, an analysis of individual versus 
group assistance is presented. Next, intervention and tutoring goals are assessed using a teacher 
rating scale implemented by Title IX staff, and by analyzing student grades directly. 

Service Method: Group versus Individual Tutoring 

Students may be served in groups or individually. A higher percentage of young students 
received group instruction than older students. By design, the Homework Center provides 
mostly individual assistance. Students attend at their own or their parents’ convenience. 



Students Receiving Group Instruction 




Elementary 


Middle 


High 


Homework 








Center 



School Level 
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Teacher Ratine Scale Analysis 

Teachers may have developed more than one goal for each student. At the three levels plus the 
homework site. Title IX staff developed 2172 goals for 1329 students. The table below shows 
the average number of goals per student by school level. 





Elementary 

School 


Middle 

School 


High 

School 


Homework 

Center 


# of Students 
# of Goals 
Average # 
Goals/Student 


377 

500 

1.33 


370 

791 

2.14 


450 

658 

1.46 


200 

223 

1.12 



Intervention and tutoring goals are combined in the tables below because they were not coded 
separately. Additionally, the varying number of goals per student required analyzing only the 
first goal for each student. Students who had one or more semesters of teacher rating scale 
scores were included. Additionally, students from each quarter were grouped to reveal the 
percentages shown in the tables. The teacher rating scale was designed to represent A through F 
grades or their symbolic counterparts (elementary schools use symbols that correspond to letter 
grade). A score of one is the lowest grade and five represents the highest grade. The percent of 
student’s ratings that met the project goal are shown in the shaded areas. The percent of students 
showing an increase in ratings pre to post are noted with an asterisk. 



Elementary Schools 

Students in elementary school maintained or improved their personal objective 94. 1% of the 
time. 



POST Quarter Teacl 


tier Rating 


PRE Quarter 
Teacher Rating 


1 


2 


3 


4 


5 


1 


4.8% 


8.5%* 


5.3%* 


1.9%* 


1.3%* 


2 


0.3% 


6.4% 


14.3%* 


3.4%* 


2.4%* 


3 


0.3% 




7.4% 


19.4%* 


0.8%* 


4 






0.5% 


7.2% 


12.7%* 


5 










3.2% 
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Middle Schools 

Students in middle school maintained or improved their personal objective 94.9% of the time. 



POST Quarter Teacher Rating 



PRE Quarter Teacher 
Rating 


1 


2 


3 


4 


5 


1 


3.0% 


9.5%* 


5.1%* 


6.8%* 


9.2%* 


2 


0.3% 


2.4% 


13.5%* 


5.1%* 


1.4%* 


3 




0.3% 


2.7% 


13.8%* 


1.1%* 


4 




0.3% 


0.3% 


18.1% 


4.3%* 


5 




0.3% 


0.3% 


0.3% 


2.2% 



High Schools 

Students in high school maintained or improved their personal objective 90.5% of the time. 



POST Quarter Teacher Rating 



PRE Quarter Teacher 
Rating 


1 


2 


3 


4 


5 


1 


3.6% 


5.1%* 


8.0%* 


0.4%* 




2 


3.1% 


7.1% 


2.4%* 


1.8%* 


0.4%* 


3 


0.4% 


0.9% 


14.2% 


12.7%* 


2.4%* 


4 


0.4% 


0.2% 


0.7% 


16.4% 


8.4%* 


5 








0.2% 


10.9% 



Homework Center 

Students participating in the homework center maintained or improved their personal objective 
99% of the time. 



POST Quarter Teacher Rating 



PRE Quarter Teacher 
Rating 


1 


2 


3 


4 


5 


1 


1.0% 


2.5%* 








2 




5.0% 


44.5%* 






3 






13.5% 


22.5%* 




4 








9.5% 




5 










1.5% 
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Each school level surpassed its stated goal, and the homework center appeared to be most 
effective. In this analysis, assistance at elementary and middle school appears to be slightly 
more effective than at high school. The homework center’s effectiveness may be a reflection of 
parent involvement since students must be transported to the homework center. Additionally, 
many students who availed themselves of the Homework Center also received assistance at 
school. In effect, the students at the homework center were often getting twice the assistance 
that other students received. 



Grade Point Average Analysis 

It was possible to see if grades as reported by project staff were affected by tutoring or 
intervention activities. The table below shows the percentage of students who had intervention 
or academic goals and whose grades improved. Some students’ records did not show all four 
quarters of data. Such missing data was thought to be due to student mobility. Therefore, 
improvement was defined as any increase from the first quarter with a recorded grade to the last 
quarter with a recorded grade. Interventions in elementary and middle school appear to improve 
more students’ grades than intervention in high school. 

Percent of student with tutoring or intervention goals w hose grades improved. 



Elementary 

Schools 


Middle 

Schools 


High 

Schools 


Home Work 
Center 


48.1 


38.5 


13.4 


33.3 
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III. “Provide for student participation in cultural, or social, or academic activities to 
good citizenship, community service, or personal responsibility” 

To meet this goal a variety of activities are planned at the school level. Field trips to Pueblo 
feast days and community service projects are examples. Other district activities include 
American Indian Knowledge Quest and Indian Week activities. 

The activity questionnaire was filled out by 387 students. Students were asked if they felt they 
had benefited from the activity. Students responded yes 85% of the time. Students left the 
question blank or answered don’t know 7% of the time, while 8% responded no. Responses to 
the explanation of the ratings were analyzed if the response was no. All but one of the no 
responses came from two schools. The students responding indicated a lack of engagement in 
the activity. Next, students were asked to rate how beneficial the activity was. Students rated 
the activity as beneficial (2 or above as defined by the grant application) 99.5% of the time. 
Fourteen students left the rating blank or wrote in don’t know. 



Responses to Student Questionnaire 



57.9 
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IV. “Provide for parent participation in cultural, or social or academic activities” 

Parents were invited to participate in social, cultural, and academic activities through 
announcements in newsletters, letters sent home with students, and announcements at meetings 
of parents of Indian students. 

There were only 1 8 responses. One respondent left blank the question of whether the students 
benefited from the activity, and another left the rating blank. Parents were asked if they thought 
activities were beneficial to students and to rate how beneficial these activities were. Eighty 
eight percent of the respondents indicated they thought the activity was beneficial to the students, 
and parents rated the activity as beneficial (2 or above) 94. 1 % of the time. 



Responses to Parent Survey 



s 

<u 

o 

u 



Ph 




The small number of survey responses makes this goal difficult to interpret. Low response rate 
is commonly due to poor survey instruments or lack of vigilance at collecting surveys. The 
small number of surveys may indicate low parent participation in the activities. It is not known 
how many parents actually participated in activities. Paperwork load for site teachers may be 
minimized by simply asking for an annual report of the titles of each activity and for the number 
of parents who participated in each activity. Once parent participation reaches sufficiently large 
numbers, surveys may be used to inform planning and improve services to students. 
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V. 



“Provide professional development training activities relating to American Indian 
students for school site administrators and classroom teachers” 

Staff development activities are planned annually based on a needs assessment survey of Native 
American students and their parents. The results indicated the continued need for the district’s 
employees to demonstrate a better understanding and knowledge of the sensitive cultural issues 
that impact student educational progress and outcomes. Staff development courses are designed 
to meet this need, and are offered to school staffs at sites where eligible students are enrolled. 
Topics covered in staff development courses include Native American students’ cultural 
uniqueness, traditions of relations with the dominant culture, motivational style, and learning 
styles of Native American students. 

One course was offered at each of 1 1 sites, and 220 APS staff participated. Evaluation forms 
were completed by 136 of those participants (62%). Participants reported 83% of the time that 
the course was helpful or very helpful. Participants reported 8% of the time that the course was 
of little benefit or no benefit. Almost 9 % of the participants reported that the course was of 
moderate benefit. The measure of improvement as stated in the grant application was a score of 
2 or better on the beneficial rating scale. Respondents indicated 98% of the time that the 
professional development was somewhat beneficial. 



Professional Development 
Outcome Survey 1997-98 




No Benefit Little Benefit Moderate Benefit Helpful Very Helpful 



Course Satisfaction 
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CONCLUSIONS 



4 



Maintaining a relational database will minimize record keeping for Title IX/Johnson O’Malley 
staff members, will facilitate the evaluation process, and will ultimately yield information that 
has more instructional usefulness. Nevertheless, general analysis indicates success in every goal 
area. Since assistance at elementary school shows the best results, Title IX staff may discuss the 
apparent benefits of early assistance and its relation to students who later drop out. It may be 
that early assistance reduces the number of dropouts at later grades. 

In past years, Title IX/Johnson O’Malley staff members have used tutoring assistance to describe 
program services. As the Title IX program grew and increased the amount of group services 
provided, the term tutoring was not applicable because the term implied one-to one assistance. 
Title IX/Johnson O’Malley staff members found the term literacy skill building to be a more 
accurate description of services. The current project faces another potential confusion point with 
the term academic assistance. Title IX/Johnson O’Malley staff members use the term academic 
assistance to describe skills identified in the District Core Curriculum Scope and Sequence as 
process skills. Aligning terms in student goals to district curriculum guides may enhance 
communication between program and school staff. 

Students reported 99.5% of the time that the social, cultural and academic activities were 
beneficial. Title IX/Johnson O’Malley staff members may wish to analyze the responses from 
the two schools that contributed the dissenting views for the purpose of staff development. 

Because of the low numbers of survey responses parent participation is difficult to evaluate. 

Title IX/Johnson O’Malley staff members may concentrate efforts on describing parent 
participation in other ways. 

Professional development efforts may require further modifications because of its successes. As 
training recipients become increasingly sophisticated about the subject matter, it is more difficult 
to satisfy their desire to leam more. Adding intermediate and advanced professional 
development classes may enable Title IX/Johnson O’Malley staff members to meet the needs of 
various levels of teacher experience. 
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